Add generation-safe prepare activation contract - #1583
Conversation
- Separate control traffic from two identified task frames - Accept a queued frame while one sequential child executor is active - Bound parent dispatch capacity and fill every available frame - Retire publish ordering across every pre-publish failure path - Reject oversized argument blobs before publishing mailbox state - Preserve single-frame fallbacks on unsupported local backends - Keep failure-path tests and host-buffer cleanup terminal-safe - Release the GIL around native execution so admission remains live
📝 WalkthroughWalkthroughThe change adds generation-safe pipeline leases, whole-run FIFO admission, prepared endpoint activation, and two-frame local mailbox execution. Run-aware scheduling, Python bindings, chip startup wiring, mailbox layouts, and unit/end-to-end tests are updated accordingly. ChangesPipeline admission and prepared execution
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/common/hierarchical/orchestrator.h`:
- Around line 132-134: Integrate can_dispatch_run into the production
run-admission/dispatch path so every run is checked against the documented
lease-ownership gate before being dispatched. Locate the scheduler method that
admits or starts runs and invoke can_dispatch_run with the candidate run_id,
rejecting or deferring runs that fail the check; leave active_run_id and
preparable_run_id unchanged.
In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 305-315: Prevent failed dispatches from leaving an unretiable
publish sequence. In the dispatch preparation flow, validate the endpoint,
prepared slot, and activation capability before calling
WorkerThread::enqueue_dispatch; for failures that can still occur after ID
allocation in dispatch_process or run_prepared_with_activation, add and invoke a
WorkerThread sequence-cancellation/poisoning hook so the allocated ID advances
or is retired and subsequent dispatches do not block.
- Around line 328-338: Update WorkerThread::stop and the activation-wait path to
synchronize shutdown_ with activation_mu_, the mutex used by the activation_cv_
predicate. Set or publish shutdown_ while holding activation_mu_ before calling
activation_cv_.notify_all(), and ensure the waiter reads it under that same lock
so shutdown cannot be missed and stop() can join all threads.
In
`@tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py`:
- Around line 129-133: Drop the loop-variable tensor references before freeing
host buffers in both cleanup blocks: update
tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py
lines 129-133 and
tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py lines
178-182 to clear the respective buffer/tensor loop bindings alongside the named
locals, then run the existing free_host_buffer sweep.
- Around line 115-119: Add a short sleep inside the polling loop around the
state reads in the worker async endpoint test, after each unsuccessful poll, so
the loop yields the GIL while retaining the existing deadline and
state-detection behavior.
In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 684-697: Replace the unbounded orchestrator.wait_run calls after
the readiness assertions with the bounded wait_run_for variant, applying the
existing test timeout convention to both first_run and second_run. Preserve the
release_normal cleanup and ensure the test reports a failure rather than hanging
when readiness was not achieved.
- Around line 575-587: Add steady-clock deadlines to both simulated-child
polling loops in the thread lambda, covering waits for PREPARE_READY and
ACTIVATE; exit or fail cleanly when either deadline expires. Also bound the
caller’s wait for completion so child.join() and caller.join() cannot hang when
readiness or done never occurs, while preserving the existing state transitions
and assertions.
In `@tests/ut/py/test_worker/test_host_worker.py`:
- Around line 1620-1625: Update the test’s finally block to join the submitter
thread before closing or unlinking shared resources, matching the cleanup
pattern in test_submit_close_race_accepts_and_drains_admitted_run. Ensure the
join occurs even when the test body fails, so submitter has exited before
hw.close(), state_shm.close(), and state_shm.unlink().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1b1a4a18-1617-45e9-bef5-e2d973cedbfd
📒 Files selected for processing (24)
docs/task-flow.mdpython/bindings/task_interface.cpppython/bindings/worker_bind.hpython/simpler/task_interface.pypython/simpler/worker.pysrc/common/hierarchical/orchestrator.cppsrc/common/hierarchical/orchestrator.hsrc/common/hierarchical/scheduler.cppsrc/common/hierarchical/scheduler.hsrc/common/hierarchical/types.cppsrc/common/hierarchical/types.hsrc/common/hierarchical/worker.cppsrc/common/hierarchical/worker.hsrc/common/hierarchical/worker_manager.cppsrc/common/hierarchical/worker_manager.hsrc/common/worker/pipeline_slot_pool.htests/st/a2a3/host_build_graph/worker_async_endpoint/kernels/orchestration/long_vector_orch.cpptests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.pytests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.pytests/ut/cpp/hierarchical/test_orchestrator.cpptests/ut/cpp/hierarchical/test_pipeline_contract.cpptests/ut/cpp/hierarchical/test_scheduler.cpptests/ut/py/test_callable_identity.pytests/ut/py/test_worker/test_host_worker.py
| bool can_dispatch_run(RunId run_id) const; | ||
| RunId active_run_id() const; | ||
| RunId preparable_run_id() const; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -nP --type=cpp --type=cc -C3 '\bcan_dispatch_run\s*\(' .
fd -e h -e hpp -e cpp -e cc --exec rg -nP -C2 '\bcan_dispatch_run\b' {} \;Repository: hw-native-sys/simpler
Length of output: 1519
🏁 Script executed:
#!/bin/bash
set -u
echo "Tracked C/C++ files mentioning can_dispatch_run:"
git ls-files '*.[ch]pp' '*.[ch]xx' '*.[ch]' |\
rg '(^|/)(src/|test|tests|include)/' | xargs -r -d '\n' grep -n -C 2 '\bcan_dispatch_run\b' || true
echo
echo "Direct references with file context:"
rg -n -C 3 '\bcan_dispatch_run\b' . || true
echo
echo "Relevant orchestrator declarations/definitions:"
fd -e h -e hpp -e cpp -e cc 'orchestrator\.(h|cpp|hxx|hpp)$' .Repository: hw-native-sys/simpler
Length of output: 6336
Wire can_dispatch_run into the run-admission path.
can_dispatch_run is only called by tests and is not used in production dispatch/admission code, so the documented lease-ownership admission gate is not enforced in the scheduler path yet.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/common/hierarchical/orchestrator.h` around lines 132 - 134, Integrate
can_dispatch_run into the production run-admission/dispatch path so every run is
checked against the documented lease-ownership gate before being dispatched.
Locate the scheduler method that admits or starts runs and invoke
can_dispatch_run with the candidate run_id, rejecting or deferring runs that
fail the check; leave active_run_id and preparable_run_id unchanged.
| void WorkerThread::enqueue_dispatch(WorkerDispatch d) { | ||
| uint32_t previous = inflight_.fetch_add(1, std::memory_order_acq_rel); | ||
| if (previous >= capacity_) { | ||
| inflight_.fetch_sub(1, std::memory_order_acq_rel); | ||
| throw std::logic_error("WorkerThread::dispatch: endpoint capacity exceeded"); | ||
| } | ||
| d.dispatch_id = next_dispatch_id_.fetch_add(1, std::memory_order_relaxed); | ||
| std::lock_guard<std::mutex> lk(mu_); | ||
| queue_.push(d); | ||
| cv_.notify_one(); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
A dispatch that fails before run_two_frame permanently stalls the publish sequence.
dispatch_id is allocated here, but next_publish_id_ is only advanced inside run_two_frame (retire_publish_sequence). Any dispatch that dies earlier — dispatch_process throwing on a null endpoint or an invalid prepared slot (lines 414-417), or run_prepared_with_activation rejecting the capability (line 594) — consumes an id and never retires it, and none of those paths poison the endpoint. Every subsequent dispatch then waits at lines 737-739 for a sequence number that will never be published, with no timeout.
Simplest fix: validate before allocating the id, and have WorkerThread retire/poison the sequence when the endpoint never got the dispatch — e.g. move the prepared-slot and endpoint validation ahead of enqueue_dispatch, and make the endpoint expose a "cancel published id" hook used on the throw paths in dispatch_process.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/common/hierarchical/worker_manager.cpp` around lines 305 - 315, Prevent
failed dispatches from leaving an unretiable publish sequence. In the dispatch
preparation flow, validate the endpoint, prepared slot, and activation
capability before calling WorkerThread::enqueue_dispatch; for failures that can
still occur after ID allocation in dispatch_process or
run_prepared_with_activation, add and invoke a WorkerThread
sequence-cancellation/poisoning hook so the allocated ID advances or is retired
and subsequent dispatches do not block.
| void WorkerThread::stop() { | ||
| { | ||
| std::lock_guard<std::mutex> lk(mu_); | ||
| shutdown_ = true; | ||
| } | ||
| cv_.notify_all(); | ||
| if (thread_.joinable()) thread_.join(); | ||
| activation_cv_.notify_all(); | ||
| for (auto &thread : threads_) { | ||
| if (thread.joinable()) thread.join(); | ||
| } | ||
| threads_.clear(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Shutdown can hang forever in the activation wait (lost wakeup → stop() never joins).
shutdown_ is stored under mu_ only, while the activation waiter (lines 435-438) blocks on activation_cv_ under activation_mu_ with a predicate that reads shutdown_. A waiter that has evaluated the predicate as false but has not yet registered on the CV misses this notify_all(), and the wait has no timeout — stop() then blocks in join() forever. The state change must be published under the same mutex the waiter uses.
🔒️ Proposed fix
void WorkerThread::stop() {
{
std::lock_guard<std::mutex> lk(mu_);
shutdown_ = true;
}
cv_.notify_all();
- activation_cv_.notify_all();
+ {
+ // Publish shutdown under activation_mu_ so a waiter between predicate
+ // evaluation and blocking cannot miss this notification.
+ std::lock_guard<std::mutex> lk(activation_mu_);
+ }
+ activation_cv_.notify_all();
for (auto &thread : threads_) {
if (thread.joinable()) thread.join();
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| void WorkerThread::stop() { | |
| { | |
| std::lock_guard<std::mutex> lk(mu_); | |
| shutdown_ = true; | |
| } | |
| cv_.notify_all(); | |
| if (thread_.joinable()) thread_.join(); | |
| activation_cv_.notify_all(); | |
| for (auto &thread : threads_) { | |
| if (thread.joinable()) thread.join(); | |
| } | |
| threads_.clear(); | |
| void WorkerThread::stop() { | |
| { | |
| std::lock_guard<std::mutex> lk(mu_); | |
| shutdown_ = true; | |
| } | |
| cv_.notify_all(); | |
| { | |
| // Publish shutdown under activation_mu_ so a waiter between predicate | |
| // evaluation and blocking cannot miss this notification. | |
| std::lock_guard<std::mutex> lk(activation_mu_); | |
| } | |
| activation_cv_.notify_all(); | |
| for (auto &thread : threads_) { | |
| if (thread.joinable()) thread.join(); | |
| } | |
| threads_.clear(); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/common/hierarchical/worker_manager.cpp` around lines 328 - 338, Update
WorkerThread::stop and the activation-wait path to synchronize shutdown_ with
activation_mu_, the mutex used by the activation_cv_ predicate. Set or publish
shutdown_ while holding activation_mu_ before calling
activation_cv_.notify_all(), and ensure the waiter reads it under that same lock
so shutdown cannot be missed and stop() can join all threads.
| while not run.done and time.monotonic() < deadline: | ||
| states = [_mailbox_load_i32(addr) for addr in frame_state_addrs] | ||
| if _TASK_ACTIVE in states and _TASK_ACCEPTED_STATE in states: | ||
| saw_active_and_accepted = True | ||
| break |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Add a small sleep to the polling loop.
This loop spins without yielding for up to 10s, holding the GIL almost continuously in the same process that runs the parent-side admission/mailbox threads. That both burns a core and can delay the very transition the test is trying to observe, making saw_active_and_accepted flaky.
🔁 Proposed fix
while not run.done and time.monotonic() < deadline:
states = [_mailbox_load_i32(addr) for addr in frame_state_addrs]
if _TASK_ACTIVE in states and _TASK_ACCEPTED_STATE in states:
saw_active_and_accepted = True
break
+ time.sleep(0.001)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| while not run.done and time.monotonic() < deadline: | |
| states = [_mailbox_load_i32(addr) for addr in frame_state_addrs] | |
| if _TASK_ACTIVE in states and _TASK_ACCEPTED_STATE in states: | |
| saw_active_and_accepted = True | |
| break | |
| while not run.done and time.monotonic() < deadline: | |
| states = [_mailbox_load_i32(addr) for addr in frame_state_addrs] | |
| if _TASK_ACTIVE in states and _TASK_ACCEPTED_STATE in states: | |
| saw_active_and_accepted = True | |
| break | |
| time.sleep(0.001) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py`
around lines 115 - 119, Add a short sleep inside the polling loop around the
state reads in the worker async endpoint test, after each unsuccessful poll, so
the loop yields the GIL while retaining the existing deadline and
state-detection behavior.
| tensors.clear() | ||
| first_a = first_b = first_out = second_a = second_b = second_out = None | ||
| if run is None or run.done: | ||
| for buffer in buffers: | ||
| st_worker.free_host_buffer(buffer) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Host-buffer cleanup misses the loop-variable tensor reference in both onboard tests. Both tests allocate buffers in a for value in (...) loop that leaves buffer / tensor bound after it ends, so clearing tensors and nulling the six named locals still leaves one live torch view over the last host buffer when free_host_buffer runs — the exact case Worker.create_host_buffer warns about ("drop the tensor first").
tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py#L129-L133: also drop the loop bindings (e.g.tensor = buffer = None) before thefree_host_buffersweep.tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py#L178-L182: apply the same drop before thefree_host_buffersweep, or factor the allocate/release pair into one shared helper so the reference set has a single owner.
📍 Affects 2 files
tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py#L129-L133(this comment)tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py#L178-L182
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py`
around lines 129 - 133, Drop the loop-variable tensor references before freeing
host buffers in both cleanup blocks: update
tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py
lines 129-133 and
tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py lines
178-182 to clear the respective buffer/tensor loop bindings alongside the named
locals, then run the existing free_host_buffer sweep.
| std::thread child([&] { | ||
| char *frame = mailbox.data() + MAILBOX_FRAME_SIZE; | ||
| auto *state = reinterpret_cast<volatile int32_t *>(frame + MAILBOX_OFF_STATE); | ||
| while (__atomic_load_n(state, __ATOMIC_ACQUIRE) != static_cast<int32_t>(MailboxState::PREPARE_READY)) { | ||
| std::this_thread::sleep_for(std::chrono::microseconds(50)); | ||
| } | ||
| __atomic_store_n(state, static_cast<int32_t>(MailboxState::BACKEND_READY), __ATOMIC_RELEASE); | ||
| while (__atomic_load_n(state, __ATOMIC_ACQUIRE) != static_cast<int32_t>(MailboxState::ACTIVATE)) { | ||
| std::this_thread::sleep_for(std::chrono::microseconds(50)); | ||
| } | ||
| __atomic_store_n(state, static_cast<int32_t>(MailboxState::TASK_ACTIVE), __ATOMIC_RELEASE); | ||
| __atomic_store_n(state, static_cast<int32_t>(MailboxState::TASK_DONE), __ATOMIC_RELEASE); | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
The simulated child spins without a deadline, so a regression hangs instead of failing.
If the endpoint stops publishing PREPARE_READY (or ACTIVATE), the child loops forever and child.join() at Line 621 never returns; likewise a done that never becomes ready leaves caller.join() at Line 620 blocked. The EXPECT_* assertions record the failure, but the binary hangs before reporting it. Bound both spin loops with a steady-clock deadline.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 575 - 587, Add
steady-clock deadlines to both simulated-child polling loops in the thread
lambda, covering waits for PREPARE_READY and ACTIVATE; exit or fail cleanly when
either deadline expires. Also bound the caller’s wait for completion so
child.join() and caller.join() cannot hang when readiness or done never occurs,
while preserving the existing state transitions and assertions.
| auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(3); | ||
| while ((!endpoint_ptr->normal_running() || !endpoint_ptr->backend_ready()) && | ||
| std::chrono::steady_clock::now() < deadline) { | ||
| std::this_thread::sleep_for(std::chrono::milliseconds(1)); | ||
| } | ||
| EXPECT_TRUE(endpoint_ptr->normal_running()); | ||
| EXPECT_TRUE(endpoint_ptr->backend_ready()); | ||
| EXPECT_FALSE(endpoint_ptr->activated()); | ||
| EXPECT_EQ(orchestrator.active_run_id(), first_run); | ||
| EXPECT_EQ(orchestrator.preparable_run_id(), second_run); | ||
|
|
||
| endpoint_ptr->release_normal(); | ||
| orchestrator.wait_run(first_run); | ||
| orchestrator.wait_run(second_run); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Unbounded wait_run after a failed readiness expectation hangs the suite.
If the 3s loop at Lines 684-688 expires (activation never staged), the EXPECT_TRUEs fail but orchestrator.wait_run(first_run) / wait_run(second_run) then block forever. Prefer the bounded wait_run_for (as used elsewhere) so this test fails rather than stalls CI.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 684 - 697, Replace
the unbounded orchestrator.wait_run calls after the readiness assertions with
the bounded wait_run_for variant, applying the existing test timeout convention
to both first_run and second_run. Preserve the release_normal cleanup and ensure
the test reports a failure rather than hanging when readiness was not achieved.
| finally: | ||
| _set_flag(state_buf, 4, 1) | ||
| _set_flag(state_buf, 12, 1) | ||
| hw.close() | ||
| state_shm.close() | ||
| state_shm.unlink() |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Join submitter in the finally block.
If the body fails before Line 1615 (e.g. the third_callback.wait(3.0) assertion), the submitter thread is still parked inside hw.submit, and the cleanup then runs hw.close() and state_shm.unlink() underneath it — a close-vs-submit race plus a leaked non-daemon thread that outlives the test. Other tests in this class (e.g. test_submit_close_race_accepts_and_drains_admitted_run) join their submitter in finally.
🧹 Proposed fix
third_callback = threading.Event()
third_result: dict[str, RunHandle] = {}
+ submitter = None
hw = Worker(level=3, num_sub_workers=2) finally:
_set_flag(state_buf, 4, 1)
_set_flag(state_buf, 12, 1)
+ if submitter is not None:
+ submitter.join(5.0)
hw.close()
state_shm.close()
state_shm.unlink()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| finally: | |
| _set_flag(state_buf, 4, 1) | |
| _set_flag(state_buf, 12, 1) | |
| hw.close() | |
| state_shm.close() | |
| state_shm.unlink() | |
| third_callback = threading.Event() | |
| third_result: dict[str, RunHandle] = {} | |
| submitter = None | |
| hw = Worker(level=3, num_sub_workers=2) | |
| ... | |
| finally: | |
| _set_flag(state_buf, 4, 1) | |
| _set_flag(state_buf, 12, 1) | |
| if submitter is not None: | |
| submitter.join(5.0) | |
| hw.close() | |
| state_shm.close() | |
| state_shm.unlink() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/ut/py/test_worker/test_host_worker.py` around lines 1620 - 1625, Update
the test’s finally block to join the submitter thread before closing or
unlinking shared resources, matching the cleanup pattern in
test_submit_close_race_accepts_and_drains_admitted_run. Ensure the join occurs
even when the test body fails, so submitter has exited before hw.close(),
state_shm.close(), and state_shm.unlink().
1 participant
Summary
PREPARE_READY -> BACKEND_READY -> ACTIVATElocal mailbox transitionsStack
This PR is stacked on #1574. The B4a-only commit is
a264f8e530c147a67023a82cd67675cf844d3359.Merge order: #1541 -> #1574 -> this PR.
Scope boundary
This PR establishes only the common scheduler/endpoint contract. It does not implement HBG epoch banks, TMR prepared state, or simulation behavior. The follow-up HBG PR will opt the production endpoint into this capability.
Validation
883 passed, 2 skipped66/66 passedtask_20260729_025038_273983421515, exit 0task_20260729_025057_275066529418, both scenes passed